Automatic remote sensing and haptic conversion system

ABSTRACT

A system is provided that automatically converts an input into one or more haptic effects in near real-time. The system senses the input in near real-time. The system automatically converts the sensed input into one or more haptic signals in near real-time. The system generates the one or more haptic effects based on the one or more haptic signals.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/019,606, filed on Sep. 6, 2013, the specification of which is herebyincorporated by reference.

FIELD

One embodiment is directed generally to a device, and more particularly,to a device that produces haptic effects.

BACKGROUND

Haptics is a tactile and force feedback technology that takes advantageof a user's sense of touch by applying haptic feedback effects (i.e.,“haptic effects”), such as forces, vibrations, and motions, to the user.Devices, such as mobile devices, touchscreen devices, and personalcomputers, can be configured to generate haptic effects. In general,calls to embedded hardware capable of generating haptic effects (such asactuators) can be programmed within an operating system (“OS”) of thedevice. These calls specify which haptic effect to play. For example,when a user interacts with the device using, for example, a button,touchscreen, lever, joystick, wheel, or some other control, the OS ofthe device can send a play command through control circuitry to theembedded hardware. The embedded hardware then produces the appropriatehaptic effect.

Devices can be configured to coordinate the output of haptic effectswith the output of other content, such as audio, so that the hapticeffects are incorporated into the other content. For example, in agaming context, when a game is developed, an audio effect developer candevelop audio effects that are associated with the game and represent anaction occurring within the game, such as machine gun fire, explosions,or car crashes. In another context, audio input, or other types ofreal-world input, can be captured by a microphone, or other type ofsensor. A haptic effect developer can subsequently author a hapticeffect for the device, and the device can be configured to output thehaptic effect along with the other content. However, such a process isgenerally not instantaneous, as the process requires a time period forthe haptic effect developer to analyze the audio input, or other input,and author an appropriate haptic effect that can be coordinated with theaudio input, or other input.

SUMMARY

One embodiment is a system that can automatically convert an input intoone or more haptic effects in near real-time. The system can sense theinput in near real-time. The system can further automatically convertthe sensed input into one or more haptic signals in near real-time. Thesystem can further generate one or more haptic effects based on the oneor more haptic signals.

BRIEF DESCRIPTION OF THE DRAWINGS

Further embodiments, details, advantages, and modifications will becomeapparent from the following detailed description of the preferredembodiments, which is to be taken in conjunction with the accompanyingdrawings.

FIG. 1 illustrates a block diagram of a system in accordance with oneembodiment of the invention.

FIG. 2 illustrates a block diagram of a system that automaticallyconverts an input into one or more haptic effects in near real-time,according to an embodiment of the invention.

FIG. 3 illustrates a flow diagram of the functionality of an automatichaptic conversion module, according to one embodiment of the invention.

DETAILED DESCRIPTION

One embodiment is a system that can include a haptic device thatdelivers haptic feedback converted from an input, such as an audiosignal, that can be sensed at a remote location by a sensor, such as amicrophone or camera, or an array of microphones or cameras, andtransmitted wirelessly to the haptic device. The haptic feedback can beoutput within the haptic device by a haptic output device or a set ofhaptic output devices, such as an actuator or a set of actuators, whichcan be programmed to deliver haptic effects that are generated based onthe input and transmitted in near real-time. More specifically, thesystem can convert the sensed input into one or more haptic signals innear real-time, and the system can further send the one or more hapticsignals to the haptic output device(s) to generate the one or morehaptic effects. In one embodiment, the sensor can be configured tofilter out and deliver only specific frequency bands, features, or othercharacteristics of the input to the haptic device. Further, according tothe embodiment, the haptic output device(s) within the haptic device canbe tuned to extract different frequency bands, features, or othercharacteristics of the input, controlling and/or enhancing the hapticfeedback in a way to modulate the attention of the user. Thus, the usercan experience haptic feedback that accentuates certain frequency bands,features, or other characteristics of the input that is sensed at theremote location. In one embodiment, the haptic device can be a wearablehaptic device.

As understood by one of ordinary skill in the art, “near real-time,” intelecommunications and computing, refers to a time delay introduced byautomated data processing or network transmission. Thus, in anembodiment where the input is sensed by the system in near real-time,the input is sensed at the time the input is generated by a remotesource, in addition to any time delay associated with sensing the input.Further, in an embodiment where the system generates one or more hapticeffects based on the sensed input in near real-time, the system canconvert the sensed input into one or more haptic signals used togenerate the one or more haptic effects at the time the sensed input isreceived by the system, in addition to any time delay associated withconverting the sensed input into the one or more haptic signals. Incertain embodiments, there can be no time delay, and thus, in theseembodiments, “near real-time” can be identical to “real-time.”

Such a system can have several applications, according to certainembodiments of the invention. For example, the system can be utilized ina military scenario, where the system can allow for stealth remotemonitoring of specific input, such as specific auditory signals, videosignals, or other type of sensed input, in near real-time, where thesystem can deliver the monitored input regarding specific events at aremote location using a haptic output, and where the haptic output canbe delivered in a completely silent manner. The haptic output may alsodeliver enough descriptive information to service in an “alert”capacity. More specifically, the haptic output can inform an individualof specific kinds of activity that are taking place, such as humanspeech events, vehicle activity or gunfire.

As another example, the system can be utilized in a scientific scenario,where the system can allow for scientists to do remote monitoring ofbiological or geological events having a distinctive input, such as adistinct audio signal, video signal, or other type of sensed input. Forexample, the system can allow an ornithologist to use a remotemicrophone to monitor for a distinct bird call that can be transmitted,converted into a haptic output, and played on a haptic device, such as awearable haptic device. Thus, the ornithologist can be informed that acertain species of bird has returned to a nesting location, or someother instance of a certain behavior at a certain time.

As yet another example, the system can be utilized in an “immersive” orentertainment scenario, where the system can allow for real-timeinformation, such as auditory information, to be ported to a viewer andconverted to one or more haptic effects. For example, in a motocrossrace, a viewer can feel what a racer is experiencing in the event innear real-time.

FIG. 1 illustrates a block diagram of a system 10 in accordance with oneembodiment of the invention. In one embodiment, system 10 is part of amobile device, and system 10 provides an automatic haptic conversionfunctionality for the mobile device. In another embodiment, system 10 ispart of a wearable device, and system 10 provides an automatic hapticconversion functionality for the wearable device. Examples of wearabledevices include wrist bands, headbands, eyeglasses, rings, leg bands,arrays integrated into clothing, or any other type of device that a usermay wear on a body or can be held by a user. Some wearable devices canbe “haptically enabled,” meaning they include mechanisms to generatehaptic effects. In another embodiment, system 10 is separate from thedevice (e.g., a mobile device or a wearable device), and remotelyprovides the automatic haptic conversion functionality for the device.Although shown as a single system, the functionality of system 10 can beimplemented as a distributed system. System 10 includes a bus 12 orother communication mechanism for communicating information, and aprocessor 22 coupled to bus 12 for processing information. Processor 22may be any type of general or specific purpose processor. System 10further includes a memory 14 for storing information and instructions tobe executed by processor 22. Memory 14 can be comprised of anycombination of random access memory (“RAM”), read only memory (“ROM”),static storage such as a magnetic or optical disk, or any other type ofcomputer-readable medium.

A computer-readable medium may be any available medium that can beaccessed by processor 22 and may include both a volatile and nonvolatilemedium, a removable and non-removable medium, a communication medium,and a storage medium. A communication medium may include computerreadable instructions, data structures, program modules or other data ina modulated data signal such as a carrier wave or other transportmechanism, and may include any other form of an information deliverymedium known in the art. A storage medium may include RAM, flash memory,ROM, erasable programmable read-only memory (“EPROM”), electricallyerasable programmable read-only memory (“EEPROM”), registers, hard disk,a removable disk, a compact disk read-only memory (“CD-ROM”), or anyother form of a storage medium known in the art.

In one embodiment, memory 14 stores software modules that providefunctionality when executed by processor 22. The modules include anoperating system 15 that provides operating system functionality forsystem 10, as well as the rest of a mobile device in one embodiment. Themodules further include an automatic haptic conversion module 16 thatautomatically converts an input into one or more haptic effects, asdisclosed in more detail below. In certain embodiments, automatic hapticconversion module 16 can comprise a plurality of modules, where eachmodule provides specific individual functionality for automaticallyconverting an input into one or more haptic effects. System 10 willtypically include one or more additional application modules 18 toinclude additional functionality, such as Integrator™ software byImmersion Corporation.

System 10, in embodiments that transmit and/or receive data from remotesources, further includes a communication device 20, such as a networkinterface card, to provide mobile wireless network communication, suchas infrared, radio, Wi-Fi, or cellular network communication. In otherembodiments, communication device 20 provides a wired networkconnection, such as an Ethernet connection or a modem.

Processor 22 is further coupled via bus 12 to a display 24, such as aLiquid Crystal Display (“LCD”), for displaying a graphicalrepresentation or user interface to a user. The display 24 may be atouch-sensitive input device, such as a touch screen, configured to sendand receive signals from processor 22, and may be a multi-touch touchscreen.

System 10, in one embodiment, further includes an actuator 26. Processor22 may transmit a haptic signal associated with a generated hapticeffect to actuator 26, which in turn outputs haptic effects such asvibrotactile haptic effects, electrostatic friction haptic effects, ordeformation haptic effects. Actuator 26 includes an actuator drivecircuit. Actuator 26 may be, for example, an electric motor, anelectro-magnetic actuator, a voice coil, a shape memory alloy, anelectro-active polymer, a solenoid, an eccentric rotating mass motor(“ERM”), a linear resonant actuator (“LRA”), a piezoelectric actuator, ahigh bandwidth actuator, an electroactive polymer (“EAP”) actuator, anelectrostatic friction display, or an ultrasonic vibration generator. Inalternate embodiments, system 10 can include one or more additionalactuators, in addition to actuator 26 (not illustrated in FIG. 1).Actuator 26 is an example of a haptic output device, where a hapticoutput device is a device configured to output haptic effects, such asvibrotactile haptic effects, electrostatic friction haptic effects, ordeformation haptic effects, in response to a drive signal. In alternateembodiments, actuator 26 can be replaced by some other type of hapticoutput device. Further, in other alternate embodiments, system 10 maynot include actuator 26, and a separate device from system 10 includesan actuator, or other haptic output device, that generates the hapticeffects, and system 10 sends generated haptic signals to that devicethrough communication device 20.

System 10, in one embodiment, further includes a speaker 28. Processor22 may transmit an audio signal to speaker 28, which in turn outputsaudio effects. Speaker 28 may be, for example, a dynamic loudspeaker, anelectrodynamic loudspeaker, a piezoelectric loudspeaker, amagnetostrictive loudspeaker, an electrostatic loudspeaker, a ribbon andplanar magnetic loudspeaker, a bending wave loudspeaker, a flat panelloudspeaker, a heil air motion transducer, a plasma arc speaker, and adigital loudspeaker. In alternate embodiments, system 10 can include oneor more additional speakers, in addition to speaker 28 (not illustratedin FIG. 1). Further, in other alternate embodiments, system 10 may notinclude speaker 28, and a separate device from system 10 includes aspeaker that outputs the audio effects, and system 10 sends audiosignals to that device through communication device 20.

System 10, in one embodiment, further includes a sensor 30. Sensor 30can be configured to detect a form of energy, or other physicalproperty, such as, but not limited to, sound, movement, acceleration,bio signals, distance, flow, force/pressure/strain/bend, humidity,linear position, orientation/inclination, radio frequency, rotaryposition, rotary velocity, manipulation of a switch, temperature,vibration, or visible light intensity. Sensor 30 can further beconfigured to convert the detected energy, or other physical property,into an electrical signal, or any signal that represents virtual sensorinformation. Sensor 30 can be any device, such as, but not limited to,an accelerometer, an electrocardiogram, an electroencephalogram, anelectromyograph, an electrooculogram, an electropalatograph, a galvanicskin response sensor, a capacitive sensor, a hall effect sensor, aninfrared sensor, an ultrasonic sensor, a pressure sensor, a fiber opticsensor, a flexion sensor (or bend sensor), a force-sensitive resistor, aload cell, a LuSense CPS² 155, a miniature pressure transducer, a piezosensor, a strain gage, a hygrometer, a linear position touch sensor, alinear potentiometer (or slider), a linear variable differentialtransformer, a compass, an inclinometer, a magnetic tag (or radiofrequency identification tag), a rotary encoder, a rotary potentiometer,a gyroscope, an on-off switch, a temperature sensor (such as athermometer, thermocouple, resistance temperature detector, thermistor,or temperature-transducing integrated circuit), microphone, photometer,altimeter, bio monitor, camera, or a light-dependent resistor. Inalternate embodiments, system 10 can include one or more additionalsensors, in addition to sensor 30 (not illustrated in FIG. 1). In someof these embodiments, sensor 30 and the one or more additional sensorsmay be part of a sensor array, or some other type of collection ofsensors. Further, in other alternate embodiments, system 10 may notinclude sensor 30, and a separate device from system 10 includes asensor that detects a form of energy, or other physical property, andconverts the detected energy, or other physical property, into anelectrical signal, or other type of signal that represents virtualsensor information. The device can then send the converted signal tosystem 10 through communication device 20.

FIG. 2 illustrates a block diagram of a system 200 that automaticallyconverts an input into one or more haptic effects in near real-time,according to an embodiment of the invention. In accordance with theembodiment, system 200 includes remote sensing device 210 and hapticdevice 220. Remote sensing device 210 can be operably coupled to hapticdevice 220 via connection 230. In certain embodiments, connection 230represents a wireless network connection. In other alternateembodiments, connection 230 represents a wired network connection.Further, in alternate embodiments, system 200 can include one or moreadditional remote sensing devices in addition to remote sensing device210 and/or one or more additional haptic devices in addition to hapticdevice 220 (not illustrated in FIG. 2). In some embodiments, connection230 can be omitted, and sensing device 210 and haptic device 220 can behoused within a single device of system 200, where sensing device 210and haptic device 220 can be operably coupled by a bus, or othercommunication mechanism for communicating information. Further, in someembodiments, haptic device 220 can be a wearable haptic device.

According to the embodiment, remote sensing device 210 includes sensor211. Sensor 211 is configured to sense input in near real-time. Morespecifically, sensor 211 is configured to detect input that is generatedin proximity of remote sensing device 210 and to convert the detectedinput into a signal in near real-time. In an alternate embodiment, thesensing of the input is not required to be in real-time or nearreal-time. Instead, in this alternate embodiment, the input can bebuffered in one or more buffers (not illustrated in FIG. 2). In certainembodiments, the input can be an audio signal, or other type of audioinput, that includes audio data. In other alternate embodiments, theinput can be a video signal, or other type of video input, that includesvideo data. In yet other alternate embodiments, the input can be anacceleration signal, or other type of acceleration input, that includesacceleration data. In yet other alternate embodiments, the input can bean orientation signal that includes orientation data, an ambient lightsignal that includes ambient light data, or another type of signal thatcan be sensed by a sensor. In certain embodiments, sensor 211 can beidentical to sensor 30 of FIG. 1.

In embodiments where the input is an audio signal, sensor 211 can be amicrophone, or some other type of device configured to sense an audiosignal. In embodiments where the input is a video signal, sensor 211 canbe a camera or some other type of device configured to sense a videosignal. In embodiments where the input is an acceleration signal, sensor211 can be an accelerometer or some other type of device configured tosense an acceleration signal. In embodiments where the input is amovement signal, sensor 211 can be a pedometer or some other type ofdevice configured to sense a movement signal. In embodiments where theinput is a temperature signal, sensor 211 can be a temperature sensor orsome other type of device configured to sense a temperature signal.Further, in certain embodiments, remote sensing device 210 can includeone or more additional sensors, in addition to sensor 211 (notillustrated in FIG. 2). In some of these embodiments, sensor 211 and theone or more additional sensors may be part of a sensor array (such as amicrophone array).

In accordance with the embodiment, remote sensing device 210 optionallyincludes filter 212. Filter 212 is configured to filter the sensedinput. More specifically, filter 212 is configured to filter out certainfrequency bands, features, or other characteristics of the sensed input.For example, where the sensed input is an audio signal, or other type ofaudio input, filter 212 can filter out characteristics of the sensedinput that represent background sound, so that the only remainingcharacteristics of the sensed input represent speech or some other typeof sound. As an alternate example, in a scenario where speech isconstantly detected, filter 212 can filter out characteristics of thesensed input that represent speech, so that the only remainingcharacteristics of the sensed input represent background sound. As yetanother example, where the sensed input is another type of input, filter212 can filter out characteristics of the sensed input so as to removeevents of small magnitude, and so that the only remainingcharacteristics of the sensed input represent events of a significantmagnitude, such as environment changes, frequency transitions, etc. Incertain embodiments, filter 212 can be a high-pass filter. Further, inalternate embodiments, filter 212 can be omitted from remote sensingdevice 210, and the corresponding filtering can be omitted.

According to the embodiment, remote sensing device 210 further includestransmitter 213. Transmitter 213 is configured to transmit the sensedinput to haptic device 220 using connection 230. In embodiments whereconnection 230 is a wireless connection, transmitter 213 can be awireless transmitter configured to wirelessly transmit the sensed inputto haptic device 220 using the wireless connection. In alternateembodiments where connection 230 is a wired connection, transmitter 213can be a wired transmitter configured to transmit the sensed input tohaptic device 220 using the wired connection. In alternate embodiments,transmitter 213 can be omitted from remote sensing device 210, and thecorresponding transmitting can be omitted.

According to the embodiment, haptic device 220 includes receiver 221.Receiver 221 is configured to receive the sensed input that istransmitted from remote sensing device 210 using connection 230. Inembodiments where connection 230 is a wireless connection, receiver 221can be a wireless receiver configured to wirelessly receive the sensedinput from remote sensing device 210 using the wireless connection. Inalternate embodiments where connection 230 is a wired connection,receiver 221 can be a wired receiver configured to receive the sensedinput from remote sensing device 210 using the wired connection. Inalternate embodiments, receiver 221 can be omitted from haptic device210, and the corresponding receiving can be omitted.

In accordance with the embodiment, haptic device 220 further includescontroller 222. Controller 222 is configured to automatically convertthe sensed input into one or more haptic signals in near-real time. Morespecifically, controller 222 is configured to receive the sensed inputfrom receiver 221 and to automatically convert the received sensed inputinto one or more haptic signals in near real-time. In certainembodiments, this means that the sensed input is directly converted intothe one or more haptic signals without storing the sensed input in amemory, or some other type of storage. In an alternate embodiment, theconversion of the received sensed input into the one or more hapticsignals is not required to be in real-time or near real-time, and caninvolve storing the sensed input in a memory, or some other type ofstorage.

Controller 222 can use any haptic conversion algorithm that is known toone or ordinary skill in the relevant art to convert the sensed inputinto the one or more haptic signals. Example haptic conversionalgorithms are described in the following patents or patent applications(all of which are hereby incorporated by reference in their entirety):U.S. Pat. No. 7,979,146; U.S. Pat. No. 8,000,825; U.S. Pat. No.8,378,964; U.S. Pat. App. Pub. No. 2011/0202155; U.S. Pat. App. Pub. No.2011/0215913; U.S. Pat. App. Pub. No. 2012/0206246; U.S. Pat. App. Pub.No. 2012/0206247; U.S. patent application Ser. No. 13/439,241; U.S.patent application Ser. No. 13/661,140; U.S. patent application Ser. No.13/743,540; U.S. patent application Ser. No. 13/767,129; U.S. patentapplication Ser. No. 13/785,166; U.S. patent application Ser. No.13/788,487; U.S. patent application Ser. No. 13/803,778; and U.S. patentapplication Ser. No. 13/799,059.

In some embodiments, controller 222 can further identify and/or extractone or more frequency bands, features, or characteristics of the sensedinput. Examples of the characteristics can include: amplitude,frequency, duration, envelope, density, magnitude, and strength. Acharacteristic can include a numeric value, where the numeric value candefine a characteristic of the sensed input. Controller 222 can furthermodify the one or more haptic signals based on the one or moreidentified characteristics. More specifically, controller 222 can modifyone or more haptic parameters of the one or more haptic signals to matchthe one or more identified characteristics. According to the embodiment,a haptic signal can include one or more haptic parameters, where ahaptic parameter is a parameter that can define the haptic signal usedto generate a haptic effect, and thus, can also define the haptic effectto be generated. More specifically, a haptic parameter is a quantity ofa haptic effect quality, such as magnitude, frequency, duration,amplitude, strength, envelope, density, or any other kind ofquantifiable haptic parameter. According to the embodiment, a hapticeffect can be defined, at least in part, by the one or more hapticparameters of the haptic signal, where the one or more haptic parameterscan define characteristics of the haptic effect. A haptic parameter caninclude a numeric value, where the numeric value can define acharacteristic of the haptic signal, and thus, can also define acharacteristic of the haptic effect generated by the haptic signal.Examples of haptic parameters can include: an amplitude hapticparameter, a frequency haptic parameter, a duration haptic parameter, anenvelope haptic parameter, a density haptic parameter, a magnitudehaptic parameter, and a strength haptic parameter.

As an example, controller 222 can identify and/or extract one or morespecific frequency bands within an audio signal, or other type of audioinput. Controller 222 can then modify one or more haptic parameters of agenerated haptic signal, so that the haptic signal includes one or morespecific frequency bands that are identical or similar to the identifiedfrequency bands within the audio signal, or other type of audio input.As is described below in greater detail, the modified haptic signal canthen be used to generate one or more distinctive haptic effects, wherethe distinctive haptic effects can identify the audio signal/input.

As another example, controller 222 can identify and/or extract a timeand/or frequency signature from an audio input. Controller 222 can thenmodify one or more haptic parameters of a generated haptic signal, sothat the haptic signal includes a time and/or frequency signature thatis identical or similar to the identified time and/or frequencysignature within the audio input. As is described below in greaterdetail, the modified haptic signal can then be used to generate one ormore distinctive haptic effects, where the distinctive haptic effectscan identify the audio input. In one particular example, the audio inputmay be a type of bird call, and the time and/or frequency signaturewithin the bird call can uniquely identify the type of bird call. Thus,the modified haptic signal may be used to generate a haptic effect thatidentifies the type of bird call based on the corresponding time and/orfrequency signature of the modified haptic signal. In another particularexample, the audio input may include human activity and non-humanactivity (such as human speech and vehicle noise), and the identifiedcharacteristics within the audio input may uniquely identify the humanactivity as distinguished from the non-human activity. Thus, themodified haptic signal may be used to generate a first haptic effectthat identifies the human activity, and a second haptic effect thatidentifies the non-human activity. An individual that experiences boththe first haptic effect and second haptic effect can identify that thefirst haptic effect identifies human activity, and the second hapticeffect identifies non-human activity.

Further, in an alternate embodiment, rather than automaticallyconverting the sensed input into one or more haptic signals, controller222 can select one or more haptic signals that are stored withincontroller 222 based on the one or more identified characteristics ofthe sensed input. More specifically, controller 222 can select one ormore haptic signals that include one or more haptic parameters that areidentical or most similar to the one or more identified characteristicsof the sensed input. The one or more haptic signals can be stored withina library, where the library is stored within controller 222.

According to the illustrated embodiment, controller 222 can includeprocessor 233, memory 224 and automatic haptic conversion module 225.Processor 233 may be any type of general or specific purpose processor.In certain embodiments, processor 233 can be identical to processor 22of FIG. 1. Memory 224 can be configured to store information andinstructions to be executed by processor 233. Memory 224 can becomprised of any combination of RAM, ROM, static storage such as amagnetic or optical disk, or any other type of computer-readable medium.In certain embodiments, memory 224 can be identical to memory 14 ofFIG. 1. Automatic haptic conversion module 225 can be configured toperform the aforementioned functionality for automatically converting aninput into one or more haptic effects. In certain embodiments, automatichaptic conversion module 225 can comprise a plurality of modules, whereeach module provides a specific individual portion of the aforementionedfunctionality for automatically converting an input into one or morehaptic effects. Further, in certain embodiments, automatic hapticconversion module 225 is identical to automatic haptic conversion module16 of FIG. 1.

Also according to the embodiment, haptic device 220 further includeshaptic output device 226. Controller 222 is configured to send the oneor more haptic signals to haptic output device 226. In turn, hapticoutput device 226 is configured to output one or more haptic effects,such as vibrotactile haptic effects, electrostatic friction hapticeffects, or deformation haptic effects, in response to the one or morehaptic signals sent by controller 222. In certain embodiments, hapticoutput device 226 is identical to actuator 26 of FIG. 1.

FIG. 3 illustrates a flow diagram of the functionality of an automatichaptic conversion module (such as automatic haptic conversion module 16of FIG. 1, and automatic haptic conversion module 225 of FIG. 2),according to one embodiment of the invention. In one embodiment, thefunctionality of FIG. 3 is implemented by software stored in memory orother computer-readable or tangible media, and executed by a processor.In other embodiments, the functionality may be performed by hardware(e.g., through the use of an application specific integrated circuit(“ASIC”), a programmable gate array (“PGA”), a field programmable gatearray (“FPGA”), etc.), or any combination of hardware and software.

The flow begins and proceeds to 310. At 310, input is sensed. In certainembodiments, the input can be sensed in near-real time. Further, incertain embodiments, the input can include audio data. In otherembodiments, the input can include video data. In other embodiments, theinput can include acceleration data. In other embodiments, the input caninclude another type of data. The flow then proceeds to 320.

At 320, the sensed input is filtered. In certain embodiments, 320 can beomitted. The flow then proceeds to 330.

At 330, the sensed input is transmitted to a controller. In someembodiments, the sensed input can be wirelessly transmitted to thecontroller. In certain embodiments, 330 can be omitted. The flow thenproceeds to 340.

At 340, one or more characteristics of the sensed input are identified.In certain embodiments, the one or more characteristics of the sensedinput can include at least one of: an amplitude, a frequency, aduration, an envelope, a density, a magnitude, or a strength. In certainembodiments, 340 can be omitted. The flow then proceeds to 350.

At 350, the sensed input is automatically converted into one or morehaptic signals. The sensed input can be automatically converted into theone or more haptic signals using a haptic conversion algorithm. Incertain embodiments, the sensed input can be automatically convertedinto the one or more haptic signals in near real-time. The flow thenproceeds to 360.

At 360, the one or more haptic signals are modified based on the one ormore identified characteristics of the sensed input. In certainembodiments, one or more haptic parameters of the one or more hapticsignals can be modified to match the one or more identifiedcharacteristics of the sensed input. In some of these embodiments, theone or more haptic parameters can include at least one of: an amplitudehaptic parameter, a frequency haptic parameter, a duration hapticparameter, an envelope haptic parameter, a density haptic parameter, amagnitude parameter, or a strength haptic parameter. In certainembodiments, 360 can be omitted. The flow then proceeds to 370.

At 370, one or more haptic effects are generated based on the one ormore haptic signals. In certain embodiments, the one or more hapticsignals can be sent to one or more haptic output devices to generate theone or more haptic effects. In some of these embodiments, at least oneof the one or more haptic output devices is an actuator. Further, incertain embodiments, a wearable haptic device can generate the one ormore haptic effects based on the one or more haptic signals. The flowthen ends.

Thus, according to an embodiment, a system is provided thatautomatically converts sensed input, such as audio signals, into hapticeffects, where the sensing of the input and the automatic conversion isperformed in near real-time. Thus, the system can automatically generatehaptic content from other input, such as audio signals. Thus, the systemcan make it significantly easier to create haptic content.

The features, structures, or characteristics of the invention describedthroughout this specification may be combined in any suitable manner inone or more embodiments. For example, the usage of “one embodiment,”“some embodiments,” “certain embodiment,” “certain embodiments,” orother similar language, throughout this specification refers to the factthat a particular feature, structure, or characteristic described inconnection with the embodiment may be included in at least oneembodiment of the present invention. Thus, appearances of the phrases“one embodiment,” “some embodiments,” “a certain embodiment,” “certainembodiments,” or other similar language, throughout this specificationdo not necessarily all refer to the same group of embodiments, and thedescribed features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with elements in configurations which are different thanthose which are disclosed. Therefore, although the invention has beendescribed based upon these preferred embodiments, it would be apparentto those of skill in the art that certain modifications, variations, andalternative constructions would be apparent, while remaining within thespirit and scope of the invention. In order to determine the metes andbounds of the invention, therefore, reference should be made to theappended claims.

We claim:
 1. A method of converting a sensed input signal into one ormore haptic effects, the method comprising: receiving the sensed inputsignal from a sensor; converting the sensed input signal into one ormore haptic signals; modifying a haptic parameter of the one or morehaptic signals to have substantially a same numeric value as a parameterof the sensed input signal, wherein the haptic parameter and theparameter of the sensed input signal are of a same parameter type,wherein the same parameter type comprises at least one of an amplitude,a frequency, a duration, an envelope, a density, a magnitude, a timesignature, a frequency signature, a frequency band, or a strength; andgenerating the one or more haptic effects based on the haptic parameterand the one or more haptic signals.
 2. The method of claim 1, whereinthe sensed input signal corresponds to at least one of a force, apressure, a strain, or a bend.
 3. The method of claim 1, wherein thesensor comprises a pressure sensor, wherein the pressure sensorcomprises at least one of a fiber optic sensor, a flexion sensor, a bendsensor, a force-sensitive resistor, a load cell, a LuSense CPS² 155, aminiature pressure transducer, a piezo sensor, or a strain gage.
 4. Themethod of claim 1, wherein the sensed input signal corresponds to apressure input, wherein the sensor comprises a pressure sensor sensingthe pressure input on a touch-sensitive input device.
 5. The method ofclaim 4, wherein the touch-sensitive input device comprises atouchscreen.
 6. The method of claim 1, wherein the sensed input signalcorresponds to an audio signal, wherein the sensor comprises amicrophone.
 7. The method of claim 1, wherein the sensed input signalcorresponds to a video signal, wherein the sensor comprises a camera. 8.The method of claim 1, wherein the sensed input signal corresponds to anacceleration signal, wherein the sensor comprises an accelerometer. 9.The method of claim 1, wherein the sensed input signal corresponds to amovement signal, wherein the sensor comprises a pedometer.
 10. Themethod of claim 1, wherein the sensed input signal corresponds to atemperature signal, wherein the sensor comprises a temperature sensor.11. The method of claim 1, wherein the sensed input signal correspondsto an ambient input, wherein the sensor comprises at least one of anaccelerometer, a gyroscope, a microphone, a photometer, a temperaturesensor, or an altimeter.
 12. The method of claim 1, wherein the sensedinput signal corresponds to a bio monitor input, wherein the sensorcomprises at least one of a temperature sensor, a bio monitor, anelectrocardiogram, an electroencephalogram, an electromyograph, anelectrooculogram, an electropalatograph, or a galvanic skin responsesensor.
 13. The method of claim 1, wherein the converting furthercomprises: wirelessly transmitting the sensed input signal to acontroller configured to perform the converting.
 14. The method of claim1, wherein the generating the one or more haptic effects is performed bya wearable haptic device.
 15. The method of claim 1, wherein thegenerating the one or more haptic effects further comprises: sending theone or more haptic signals to one or more haptic output devices togenerate the one or more haptic effects.
 16. A computer-readable mediumhaving instructions stored thereon that, when executed by a processor,cause the processor to convert a sensed input signal into one or morehaptic effects, the converting comprising: receiving the sensed inputsignal from a sensor; converting the sensed input signal into one ormore haptic signals; modifying a haptic parameter of the one or morehaptic signals to have substantially a same numeric value as a parameterof the sensed input signal, wherein the haptic parameter and theparameter of the sensed input signal are of a same parameter type,wherein the same parameter type comprises at least one of an amplitude,a frequency, a duration, an envelope, a density, a magnitude, a timesignature, a frequency signature, a frequency band, or a strength; andgenerating the one or more haptic effects based on the haptic parameterand the one or more haptic signals.
 17. The computer-readable medium ofclaim 16, wherein the sensed input signal corresponds to at least one ofa force, a pressure, a strain, or a bend.
 18. The computer-readablemedium of claim 16, wherein the sensor comprises a pressure sensor,wherein the pressure sensor comprises at least one of a fiber opticsensor, a flexion sensor, a bend sensor, a force-sensitive resistor, aload cell, a LuSense CPS² 155, a miniature pressure transducer, a piezosensor, or a strain gage.
 19. The computer-readable medium of claim 16,wherein the sensed input signal corresponds to a pressure input, whereinthe sensor comprises a pressure sensor sensing the pressure input on atouch-sensitive input device.
 20. A system for converting a sensed inputsignal into one or more haptic effects, the system comprising: a memoryconfigured to store a haptic conversion module; and a processorconfigured to execute the haptic conversion module stored on the memory;wherein the haptic conversion module is configured to receive the sensedinput signal from a sensor; wherein the haptic conversion module isfurther configured to convert the sensed input signal into one or morehaptic signals; wherein the haptic conversion module is furtherconfigured to modify a haptic parameter of the one or more hapticsignals to have substantially a same numeric value as a parameter of thesensed input signal, wherein the haptic parameter and the parameter ofthe sensed input signal are of a same parameter type, wherein the sameparameter type comprises at least one of an amplitude, a frequency, aduration, an envelope, a density, a magnitude, a time signature, afrequency signature, a frequency band, or a strength; and wherein thehaptic conversion module is further configured to generate the one ormore haptic effects based on the haptic parameter and the one or morehaptic signals.